In this work, we detail a novel open source library called MMLSpark that combines the flexible deep learning library Cognitive Toolkit with the distributed computing framework Apache Spark. To achieve this union, we have contributed Java Language bindings to the Cognitive Toolkit and added several new components to the Spark ecosystem. In addition, we integrate OpenCV, the popular image processing library, with Spark. We present a tool that automatically generates PySpark wrappers from any SparkML estimator, and we use this tool to expose all work to the PySpark ecosystem. Finally, we provide a large library of tools for working and developing within the Spark ecosystem. We apply this work to the automated classification of Snow Leopards from camera trap images, and provide an end-to-end solution for the non-profit conservation organization, the Snow Leopard Trust.
Publication - Flexible and Scalable Deep Learning with MMLSpark
· One min read